Handheld mobile devices, such as smartphones, commonly include video cameras that enable a user to record digital video. However, digital videos recorded on a handheld mobile device frequently include undesired motion resulting from the user's shaky or erratic hand movements during recording. Other distortions, such as wobble and skew, may occur due to a digital camera's use of a rolling shutter sensor. Further distortions may be caused by a phenomenon known as focus breathing, which involves a change in focal length that occurs when the focus distance of the lens is changed, causing the image boundary to unexpectedly shift.
Various video stabilization techniques attempt to correct for some of these problems, but known techniques suffer from various limitations. For example, image-based analysis that attempts to estimate the camera motion from the video input is not always reliable because noise, blur, and the presence of dominant objects in the video frame may cause unreliable estimates of camera motion. Some solutions operate on cinematographic cameras or may be performed in post-production operations, but such solutions are not viable for real-time operation on a mobile device due to the inherently limited computing resources of such devices. Many mobile device solutions are also unable to deliver consistent results in preview and video, leading the final video composition to look different from what a user may see while recording. For example, some mobile device video stabilization techniques use a Gaussian filter to model the motion of the camera, which does not produce high-quality results.
Therefore, it would be desirable to provide a video stabilization technique that can more accurately model the motion of the camera to remove undesired movements, rolling shutter distortion, undefined black boundaries, and focus breathing from a video to produce a high quality video on a mobile device such that both the preview view of the mobile device and the corresponding video stream are stabilized in real-time.